Introduction to Machine Learning

Chapter 15: Regularized Regression and Feature Selection

1. Introduction

An unconstrained regression will happily grow enormous coefficients to fit noise. Regularisation prevents this by adding a penalty on coefficient magnitude to the cost function, buying a reduction in variance at the price of a little bias — often a very good trade.

The choice of penalty norm determines the character of the result. Ridge (L2) shrinks coefficients smoothly toward zero without ever quite reaching it. Lasso (L1) drives coefficients exactly to zero, which makes it a feature-selection method in its own right — an embedded method, selecting features as a side effect of fitting. Elastic Net blends the two. Because Lasso blurs the line between regularisation and selection, this chapter also covers the other selection techniques specific to regression: the correlation filter and p-value based selection, and where each sits in the filter / wrapper / embedded taxonomy.

Learning Objectives

2. Theory

2.1 Regularization Motivation

Consider two candidate models that both fit the training data equally well:

Model 1 (Simple)Model 2 (Complex)
\( \hat{y} = 2x_3 + 1.4x_7 - 0.5x_9 + 4 \) \( \hat{y} = 22x_1 - 103x_2 - 14x_3 + 109x_4 - 93x_5 + 203x_6 + 87x_7 - 55x_8 + 378x_9 - 25x_{10} + 8 \)

We should prefer Model 1 — simpler, sparser, with smaller coefficients. Why are large / many coefficients problematic?

  1. Overfitting: The model memorizes training noise instead of true signal.
  2. Poor generalization: Complex models perform worse on unseen data.
  3. Sensitivity: Large coefficients amplify tiny feature variations and noise.
  4. Interpretability: 10 large coefficients are much harder to reason about than 3 small ones.

Solution: Add a regularization term to the cost function that penalizes coefficient magnitude. The result: the optimizer is forced to trade off low training error for model simplicity.

2.2 L1 and L2 Norms

Two mathematical ways to penalize coefficient magnitude. Note: only the weights \( \theta_1, \ldots, \theta_p \) are regularized — never the bias \( \theta_0 \).

L1 Norm (Sum of |·|)
L2 Norm (Sum of squares)
\[ \|\theta\|_1 = \sum_{j=1}^{p-1} |\theta_j| \]

Used in Lasso Regression. Produces sparse solutions (some coefficients become exactly zero).

\[ \|\theta\|_2^2 = \sum_{j=1}^{p-1} \theta_j^2 \]

Used in Ridge Regression. Shrinks all coefficients towards zero (none hit exactly zero).

Numerical Example: L1 vs. L2 Magnitude

Model 1: \( \hat{y} = 2x_3 + 1.4x_7 - 0.5x_9 + 4 \) (weights: 2, 1.4, −0.5; bias 4, not penalized)

Model 2: weights = [22, −103, −14, 109, −93, 203, 87, −55, 378, −25]

Notice how the L2 penalty grows quadratically for Model 2. Because the penalty rises much faster than the L1 penalty as coefficients get larger, Ridge assigns Model 2 a far higher cost than Lasso does.

2.3 Balancing Performance and Complexity: The Full Cost Function

The total loss is the sum of (a) regression error (quality of fit) and (b) a regularization term (penalty for complexity):

\[ J(\theta) = \text{MSE-error}(\theta) + \lambda \cdot \text{Regularization}(\theta) \]

The hyperparameter \( \lambda \) (lambda) controls the balance between performance and simplicity.

2.4 Ridge vs. Lasso Regression

Substituting the L2 norm into the cost function gives Ridge regression, and substituting the L1 norm gives Lasso. The choice changes how the coefficients shrink, and this is what makes Lasso usable as a feature-selection method:

AspectRidge Regression (L2)Lasso Regression (L1)
Full Cost Function \( J = \frac{1}{2m}\sum(\hat{y}_i-y_i)^2 + \lambda \sum_{j=1}^{p-1} \theta_j^2 \) \( J = \frac{1}{2m}\sum(\hat{y}_i-y_i)^2 + \lambda \sum_{j=1}^{p-1} |\theta_j| \)
Effect on coefficients Shrinks all coefficients toward zero; none set to exactly zero Shrinks and sparsifies: some coefficients become exactly zero
Feature selection Retains all features (still uses them all in predictions) Automatic feature selection (Embedded method!); zeroed-out features are dropped
Interpretability Less interpretable (all p features remain) More interpretable (sparse, fewer non-zero coefficients)
Standardization Strongly recommended (L2 penalty is very scale-sensitive) Recommended (if skipped, regularization is applied unevenly across features)

2.5 Tuning the Regularization Parameter λ

Trying to make the model perform better can make it more complex, and vice versa. \( \lambda \) is the "knob" that resolves this tension:

2.6 Elastic Net

Elastic Net combines both penalties. A hyperparameter \( \rho \in [0, 1] \) (rho) weights the L1 and L2 terms:

\[ J(\theta) = \frac{1}{2m}\sum_{i=1}^{m} (h_\theta(x_i) - y_i)^2 + \lambda \left[ \rho \sum_{j=1}^{p-1} |\theta_j| + (1-\rho) \sum_{j=1}^{p-1} \theta_j^2 \right] \]

Library notation note: In scikit-learn the mixing parameter is called l1_ratio (α in some textbooks). We use \( \rho \) here to avoid confusion with the gradient-descent learning rate α.

2.7 Feature Selection for Regression

Feature selection improves model performance, training speed, and interpretability by discarding irrelevant or redundant features. We focus on two regression-tailored methods.

Method 1: Correlation Filter (Model-Independent)

In the chapter on feature selection we saw Chi-Square, ANOVA and other filter methods, all of which assume a categorical target. For regression with a continuous target, the Pearson Correlation Coefficient is the most common filter. It measures the linear relationship between each feature and the target.

\[ r \in [-1,\ +1] \]

Like other filter methods, we can either keep the top-k columns or select columns exceeding a threshold (say \( |r| > 0.3 \)).

Note: Correlation works with one-hot encoded categorical variables, but ANOVA or Mutual Information are more statistically natural choices for purely categorical features.

Method 2: p-value-Based Selection (Embedded Method)

After training a linear regression model, we can examine the statistical significance of each coefficient via its p-value. Formally, we test the null hypothesis:

\[ H_0: \theta_j = 0 \quad \text{("feature } j \text{ has no effect on the target")} \]

Decision rule (typical):

Feature Selection Families (Big Picture)

FamilyHow it WorksExamples
Filter Select features before training, using statistical tests independent of the final model Chi-square, ANOVA, Pearson Correlation, Mutual Information
Wrapper Train the model many times with different subsets to pick the best-performing subset Forward Selection, Backward Elimination, Recursive Feature Elimination (RFE)
Embedded Feature selection happens during / as a byproduct of model training Tree-based feature importances, Lasso (this chapter!), p-value pruning

3. Interactive Examples

Example 1: λ Edge Cases

A. What model do we recover when λ = 0 in Ridge regression?

Ordinary Least Squares (OLS) / standard Linear Regression. The entire regularization term cancels out: \( \lambda = 0 \implies J = \text{MSE only} \). You'll get the same coefficients as the Normal Equation solution.

B. What happens as λ → +∞ in Lasso regression?

The regularization term dominates the loss. The optimum is to set ALL weights \( \theta_1, \ldots, \theta_p \) to exactly zero. The model then predicts only the bias (the mean y of the training set), regardless of the input features. This is an underfit model — use cross-validation to pick a finite λ!

Example 2: Ridge or Lasso?

Pick the more appropriate regularization method for each goal.

Scenario A: You have 500 features and suspect only 20 of them matter; your stakeholders want a short, human-readable list of "the drivers" to put in a report.

Lasso. Its L1 penalty will zero out many of the 480 irrelevant features, giving you the sparse, interpretable shortlist stakeholders need.

Scenario B: You have 20 carefully chosen features from domain experts; each is known to be important. You just want to dampen coefficients and avoid overfitting, without dropping any feature.

Ridge. All 20 features remain with shrunk but non-zero coefficients, respecting the domain knowledge that each feature contributes meaningfully.

Example 3: Correlation Filter Reasoning

A dataset of 8 features has Pearson correlations with the target shown below:

FeatureCorrelation (r) with Target
Age+0.04
Income+0.72
Zip-code (one-hot)−0.02
Education-Years+0.31
Height−0.08
Credit Score−0.58
Shoe Size+0.01
Family Size+0.22

Task: Apply the threshold \( |r| > 0.3 \). Which features are kept?

Kept (|r| > 0.3):
  • Income (\( r = +0.72 \)) — strong positive linear relationship
  • Education-Years (\( r = +0.31 \)) — just above threshold, positive
  • Credit Score (\( r = -0.58 \)) — moderate negative linear relationship
Dropped (low linear association): Age, Zip-code, Height, Shoe Size, Family Size (0.22 < 0.3).

Caution: Correlation only captures linear association. A strong non-linear relationship could have r ≈ 0 and would be dropped by this filter.

Example 4: Spot the p-value Interpretation Mistake

A student argues: "Since feature X's p-value is 0.08 (greater than 0.05), we have proven that X has no effect on the target whatsoever."

Mistake: Confusing "failure to reject \( H_0 \)" with "accepting \( H_0 \)." A high p-value is not proof of no effect.

Correct interpretation:

With p = 0.08, the observed data are not sufficiently unlikely under the null hypothesis \( \theta_j = 0 \). So we fail to reject \( H_0 \) at the α = 0.05 level. This does not mean the feature is definitely irrelevant — it might be a weak effect or the sample might be too small to detect it. Use domain knowledge and cross-validated performance before dropping it.

4. Numerical Solutions

Problem 1: Elastic Net Penalty Term

Weights: \( \theta = [\theta_1 = 3,\ \theta_2 = -4]^T \). λ = 0.1, ρ = 0.6.

📘 Compute the total Elastic Net penalty value

Step 1: L1 part (weighted by ρ):

\[ \rho \sum |\theta_j| = 0.6 \cdot (|3| + |-4|) = 0.6 \cdot 7 = 4.2 \]

Step 2: L2 part (weighted by 1 − ρ):

\[ (1-\rho) \sum \theta_j^2 = 0.4 \cdot (3^2 + (-4)^2) = 0.4 \cdot 25 = 10 \]

Step 3: Multiply by λ and sum:

\[ \lambda(\text{L1-part} + \text{L2-part}) = 0.1 \cdot (4.2 + 10) = \mathbf{1.42} \]

Problem 2: Ridge-Adjusted Normal Equation

Adding L2 regularization to the Normal Equation changes the closed-form solution to:

\[ \theta_{\text{ridge}} = (X^T X + \lambda I')^{-1} X^T y \]

where \( I' \) is the identity matrix but with \( I'_{00} = 0 \) (bias is not regularized). Prove / reason: "Why does adding \( \lambda I' \) guarantee invertibility, even when \( X^T X \) is singular?"

📘 Step-by-Step Reasoning

Step 1: \( X^T X \) is always positive semi-definite: for any vector v, \( v^T X^T X v = \|Xv\|^2 \ge 0 \).

Step 2: Singularity ⟺ some non-zero v exists with \( \|Xv\| = 0 \) (i.e., X has linearly dependent columns).

Step 3: Adding \( \lambda I' \) (with λ > 0 and the bias trick) to \( X^T X \) shifts every eigenvalue of the feature-submatrix by λ. The result is positive definite:

\[ v^T (X^T X + \lambda I') v = \|Xv\|^2 + \lambda \sum_{j \ge 1} v_j^2 > 0 \quad \forall v \ne 0 \]
Positive definite matrices are always invertible. ✓ This is one of Ridge's greatest practical benefits: it fixes multicollinearity / non-invertibility for free.

Problem 3: Correlation + p-value Combined Reasoning

Feature A: Pearson r = +0.02 with target, p-value = 0.01 after regression.
Feature B: Pearson r = +0.55 with target, p-value = 0.20 after regression.

📘 Step-by-Step Interpretation

Step 1: Feature A (r = 0.02, p = 0.01)

  • Marginal correlation with target is tiny → a correlation filter would drop it.
  • But in the multi-variate model, its coefficient is statistically significant (p < 0.05).
  • Interpretation: A has little univariate linear relation but helps prediction conditional on the other features (suppression / interaction effect). Keep A.

Step 2: Feature B (r = 0.55, p = 0.20)

  • Univariate correlation is strong → a correlation filter would keep it.
  • But in the full model, its coefficient is not statistically significant.
  • Likely cause: B is highly correlated with another feature already in the model (multicollinearity). The model doesn't "need" both. Consider removing B or the collinear partner (use cross-validated performance to decide).

Takeaway: Correlation and p-values answer different questions. Use both, not either one in isolation.

5. Try It Yourself

Problem 1 — Ridge vs. Lasso: Compute Penalty Values

Two models with the same 3 non-bias weights: θ = [5, 0, −5].

  1. Compute the L1 penalty term \( \sum |\theta_j| \).
  2. Compute the L2 penalty term \( \sum \theta_j^2 \).
  3. Which penalty method "dislikes" this weight vector more (i.e., produces a bigger numeric penalty)?
  1. L1 = |5| + |0| + |−5| = 10
  2. L2 = 5² + 0² + (−5)² = 25 + 0 + 25 = 50
  3. L2 (Ridge) penalizes it 5× more because it squares the large ±5 weights. L2 disproportionately hates big coefficients, which is exactly why it shrinks them.
Problem 2 — Elastic Net Corner Cases

Elastic Net penalty with λ = 0.5.

  1. If ρ = 0, what method does Elastic Net reduce to? Write its penalty expression using θ = [3, 4].
  2. If ρ = 1, what method does Elastic Net reduce to? Write its penalty expression using θ = [3, 4].
  3. For a given finite λ > 0, which of ρ = 0 vs. ρ = 1 is guaranteed to produce at least one exactly zero coefficient when there are many irrelevant features?
  1. Ridge (only L2): \( 0.5 \cdot (3^2 + 4^2) = 0.5 \cdot 25 = 12.5 \).
  2. Lasso (only L1): \( 0.5 \cdot (|3| + |4|) = 0.5 \cdot 7 = 3.5 \).
  3. ρ = 1 (Lasso). L1 geometry creates corners at the axes of the parameter space, so optima land exactly on them (some θⱼ = 0). Ridge (ρ = 0) never produces exactly zero coefficients — only shrinks them toward zero.
Problem 3 — Feature Selection Family Matching

Match each description to the correct family: (F) Filter, (W) Wrapper, (E) Embedded.

  1. "I run my Random Forest once and inspect feature_importances_ to drop unimportant features."
  2. "I rank all features by ANOVA F-value vs. the target, then keep the top 20."
  3. "I start with 0 features, keep adding the feature that most improves CV score, and stop when the score plateaus."
  4. "I fit a linear regression and drop all features with p > 0.05."
  1. (E) Embedded — selection is a byproduct of the RF training.
  2. (F) Filter — statistical scoring before any model is trained.
  3. (W) Wrapper (Forward Selection) — re-trains the model many times searching subsets.
  4. (E) Embedded — uses statistics computed during / after the regression fit.

6. Interactive Quiz

Answer all 5 questions. Click an option for instant feedback.

Your score: 0 / 5

7. Key Takeaways

  1. Two regularization norms: L1 = sum |θⱼ| → sparse solutions (Lasso). L2 = sum θⱼ² → shrinkage toward zero (Ridge). Bias θ₀ is NEVER regularized.
  2. Full cost: J = MSE + λ·Regularization. λ = 0 recovers OLS; large λ underfits; optimal λ via cross-validation.
  3. Ridge vs. Lasso: Ridge keeps all features (good when all are useful). Lasso zeros some out automatically (embedded feature selection + interpretability).
  4. Elastic Net interpolates Ridge ↔ Lasso with ρ mixing parameter. In practice, ρ = 0.5 often performs best when features are collinear.
  5. Gradient Boosting builds a sequence of trees: Tree 1 predicts the mean; Tree 2+ predict residuals and are summed with a small learning rate ν. Smaller corrections at each step = better generalization.
  6. 3 key GBM hyperparameters: n_estimators (# trees), max_depth (per-tree complexity), learning_rate ν (shrinkage). They must be tuned together via CV — they are tightly coupled.

8. Common Pitfalls

  1. Applying Lasso without standardizing features first. Feature in "dollars" (large values) gets far more L1 penalty than the same feature in "millions of dollars" (tiny values). Always standardize before any penalized regression.
  2. Tuning λ on the test set. Tuning any hyperparameter on the test set leaks information and produces inflated, misleading scores. Use cross-validation on the training set only.
  3. Forgetting that in scikit-learn, n_estimators=4 means 4 additive trees (+ the initial mean in GBM). Off-by-one errors between "number of weak learners" and "number of additive stages" are easy in exam questions.
  4. Using an aggressive learning_rate (0.5+) on large-data GBM and wondering why validation loss plateaus early. Small ν (0.01–0.1) + many trees + early stopping is almost always the winning recipe.
  5. Assuming "Lasso is always better because it does feature selection." When all features are legitimately useful, Lasso's zeroing actually hurts performance by dropping signal. Ridge (or Elastic Net with low ρ) can win here.
  6. Calling Elastic Net a "third separate norm." It's just a weighted combination of L1 and L2 — not a fundamentally new penalty shape. The ρ parameter just dials between the two known extremes.